1. Field of the Invention
The invention relates to a signal processing method for providing interpolated values between the sample values in a sampled image signal. More specifically, the invention relates to such signal processing methods for providing interpolated values between sample values of checkerboard sampling pattern.
2. Description of the Related Art
A single-chip color image sensor is employed in numerous image capturing apparatuses including digital still cameras to reduce the size and cost of the apparatus.
In the imaging plane of a single-chip color image sensor, pixels including image sensing element are arranged to two-dimension. Being overlaid with a color filter array, the pixels of the single-chip color image sensor alternate between pixels for sampling luminance component (e.g. green) and pixels for sampling one and then the other of the chrominance components (e.g. red and blue). When reconstructing an image from the sampled color image signals produced by the single-chip color image sensor, values of the luminance component and chrominance components are provided for each sample location. This is generally accomplished by some signal processing method involving linear interpolation. From the characteristic that human visual system has big sensitivity to a luminance component, it is known that the resolution of a reproduced picture relates to the resolution of a luminance component. Consequently, the resolution of a reproduced picture is greatly concerned with the way of generating interpolated luminance component.
In many digital still cameras, a single-chip color image sensor employs well known filter array pattern, the checkerboard pattern described in U.S. Pat. No. 3,971,065 issued Jul. 20, 1976 to B. E. Bayer. The checkerboard pattern is often called a Bayer array.
FIG. 1 shows the color sampling pattern of the Bayer array. The luminance sampling elements, labeled G, are separated by chrominance sampling elements, labeled R or B. In the Bayer array, luminance sampling elements, labeled G, occur at every other sampling location to provide a relatively high frequency sampling pattern which is uniform in horizontal and vertical directions. Chrominance sampling elements, labeled R or B, is surrounded by four immediately adjacent luminance sampling elements. In FIG. 1, the number in the upper part of the array expresses row number, and the number in the left side of the array expresses column number respectively.
U.S. Pat. No. 4,642,678 issued Feb. 10, 1987 to D. R. Cok shows a method for processing sampled image signals produced by a single-chip color image sensor having a checkerboard luminance sampling pattern. The patent discloses a signal processing method for producing neighboring hue values representing a hue component of the image at neighboring chrominance component sample locations as a function of a luminance value and a chrominance value at the neighboring locations, producing an interpolated hue value representing the hue component of the image at an interpolation location as a function of the neighboring hue values, and producing an interpolated chrominance value as a function of the interpolated hue value and a luminance value at the interpolating location. The signal processing method disclosed in the patent reduces color fringing in an image reproduced from the sampled image signal without introducing unwanted hue shifts. On the other hand, the patent discloses a way of generating interpolated luminance values that averages four nearest neighbors like a digital low-pass filter. Thereafter, the procedure for the interpolation of luminance values causes the decline of the frequency content of the image signal.
Moreover, U.S. Pat. No. 4,630,307 issued Dec. 16, 1986 to D. R. Cok discloses a signal processing method for providing interpolated values between sampled values in a sampled image signal produced by a single-chip color image sensor having a checkerboard luminance sampling pattern. The signal processing method disclosed in the patent is characterized by: (1) providing a plurality of different interpolation routines for producing interpolated signal values appropriate for completing a respective plurality of known geometrical image features; (2) detecting which of the known geometrical image features is represented by a neighborhood of sample values; and (3) applying the interpolation routine appropriate for completing the detected feature to the neighboring sample values to produce the interpolated signal value. In a preferred embodiment of the patent, for processing a sampled image signal from a single-chip color image sensor having a checkerboard pattern, the image features are detected in a four-sample local neighborhood. The features include an edge, a stripe, and a corner.
When the image features are detected correctly, the signal processing method disclosed in the patent can provide the interpolated signal value without errors. However, it is difficult to perform detection stably when the image signal includes large noise.
As is well known, interpolated values are generated by performing an inverse discrete cosine transform (IDCT) using a block of frequency coefficients obtained by performing a discrete cosine transform (DCT) to the blocks of original sampled values, decreasing the sampling interval from that of original sampling locations. In this case, since the inverse discrete cosine transform uses the block of frequency coefficients transformed from the original sampling values, the frequency content of the image signal is not affected by the interpolated values.
The outline of the interpolation method performing a discrete cosine transform and an inverse discrete cosine transform is explained using FIGS. 2, 3, and 4.
FIG. 2 shows locations of the original pixels in one block of 8-by-8 pixels used in a discrete cosine transform. The number “i” which shows horizontal location is zero to 7 from a left end to a right end, and the number “j” which shows perpendicular location is zero to 7 from the topmost part to the lowermost part. Here, x(i, j) is defined as the signal of the pixel located at horizontal location i and vertical location j, and F(u, v) is defined as the frequency coefficient obtained by a discrete cosine transform for horizontal frequency u and vertical frequency v, it is known that F(u, v) is given by formula 1 shown in FIG. 10, where N is the number of original pixels in a column (or a row) of a block, being 8 in FIG. 2.
According to formula 1, the value u or v can vary from zero to 7. Since the value of cosine term of F(u, V) does not change by the pixel location when u=0 (or v=0), u=0 (or v=0) means a direct current of frequency zero. The value of cosine term of F(u, v) varies from cos(PI/16) to cos(15PI/16) between i=0 and i=7 when u=1, where “PI” is defined as pi. Since that variation range corresponds to a phase change of pi exactly at the interval of one block, u=1 (or v=1) means frequency fs/16, where fs is the frequency correspond to the reciprocal of a pixel interval. Since the frequency corresponds to u=7 is 7 times the frequency corresponds to u=1, the maximum frequency which u (or v) takes in F(u, v) is 7fs/16. Consequently, the locations where F(u, v) is defined in the frequency domain are shown by white points in FIG. 3. In FIG. 3, the horizontal frequency u is expressed on a horizontal axis, the vertical frequency v is expressed on a perpendicular axis. The white points express not the values but the locations of the frequency coefficients.
Similarly, the resulting block of 8-by-8 frequency coefficients is transformed back to the spatial domain by use of an inverse discrete cosine transform given by formula 2 shown in FIG. 11.
For understanding the mathematical basis of interpolation, it is important to note that, upon replacing the integer variable, 2i and 2j, by integer i′ and j′ respectively, such that i′ (or j′)=0 to 15, there is obtained the relationship for the inverse discrete cosine transform given by formula 3 shown in FIG. 12, wherein x′(i′,j′) is an interpolated signal function. The pixel location of x′(i′, j′) generated by formula 3 is as being shown in FIG. 4. Since i′ is replaced by 2i, the signal x′(i′,0) corresponds to the pixel location i′=0 (, 2, 4, 6, 8, . . . , or 14) is equivalent to the signal x(i,0) corresponds to the pixel location i=0 (, 1, 2, 3, 4, . . . , or 7). Similarly, the signal x′(0,j′) corresponds to the pixel location j′=0 (, 2, 4, 6, 8, . . . , or 14) is equivalent to the signal x(0,j) corresponds to the pixel location j=0 (, 1, 2, 3, 4, . . . , or 7). That is, each pixel location shown in FIG. 4 with a white point is equivalent to the original pixel location shown in FIG. 2. On the other hand, each pixel location shown in FIG. 4 with black point is the additional pixel location. Since the frequency coefficients generated from the block of original pixels are used for reproduction of the interpolated values, the interpolated signal can be provided without affecting the frequency content of the image signal.
For example, U.S. Pat. No. 5,168,375 issued Dec. 1, 1992 to M. L. Reisch and M. A. Wober discloses a method for processing a field of image data samples to provide for one or more of the functions of decimation, interpolation, and sharpening that is accomplished by use of an array transform processor such as that employed in a JPEG compression system.
However, since the zero values in a block of pixels produced by the luminance signal of checkerboard pattern are represented by an inverse discrete cosine transform as zero, there is no example in which a discrete cosine transform is applied only to the original sampled values of the luminance signal.